Papers with perplexity reduction
Surface-Based Retrieval Reduces Perplexity of Retrieval-Augmented Language Models (2023.acl-short)
Copied to clipboard
| Challenge: | Augmenting language models with a retrieval mechanism has been shown to improve performance while keeping the number of parameters low. |
| Approach: | They propose to augment language models with a retrieval mechanism by replacing semantic retrieval with BM25 . they find that the model's performance is better explained by surface-level similarities, they say . |
| Outcome: | The proposed method reduces perplexity and lowers the number of parameters while keeping the number low. |
OCP: Outlier-Centric Probing for Dynamic Structured Pruning of LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Existing structured pruning methods fail to identify outlier-triggering tokens and uniform layer-wise sparsity misaligns with heterogeneous outlier distributions. |
| Approach: | They propose a framework that prioritizes capturing outlier-triggering tokens rather than reconstructing full hidden distributions. |
| Outcome: | Experiments on LLaMA2, LLama3 and OPT show that the proposed framework outperforms state-of-the-art methods and achieves 25% perplexity reduction at 1.6 speedup. |
LaCo: Layer-wise Compensation for Pruned Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for predicting performance degradations of Large Language Models (LLMs) neglect the structural distortions caused by sparsity. |
| Approach: | They propose a framework that reorients the recovery paradigm from global adaptation to hierarchical representation alignment by sequentially optimizing each layer to reconstruct the model's hidden states. |
| Outcome: | The proposed framework surpasses parameter-efficient baselines in perplexity reduction and zero-shot reasoning. |